Traffic lane marking line recognition system for vehicle

ABSTRACT

A traffic lane marking line recognition system for vehicle including a traffic lane marking line recognizing device configured to recognize at least a traffic lane marking line in broken (e.g., white) line on a road surface in a camera photographed image, and an image compositing device which composites images photographed at different time points to elongate the traffic lane marking line in the photographed image, wherein the images are composited in a processing stage in such a manner that no change will be imparted to at least a shape of the traffic lane marking line in the photographed image in a traffic lane marking line recognition processing, specifically in a processing stage prior to edge detection. With this, a distant traffic lane marking line is prevented from being chipped away when compositing the photographed images, thereby enabling accurate and unerring recognition of the traffic lane marking line.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a U.S. National phase of, and claims prioritybased on PCT/JP2005/008937 (published as WO 2005/111937 A1), which, inturn, claims priority from Japanese patent application 2004-148577,filed May 19, 2004. The entire disclosure of each of the referencedpriority documents is incorporated herein by reference.

TECHNICAL FIELD

This invention relates to a traffic lane marking line recognition systemfor vehicle.

BACKGROUND ART

Road surfaces (roads) on which vehicles drive are marked or installedwith various road markings such as white lines, yellow lines and cat'seyes constituting traffic lane marking lines. Conventionally therecognition of white lines and other such traffic lane marking lineshas, as taught by Patent Reference 1, been done by subjecting an imagephotographed or taken by image photographing means such as a CCD cameraand an image processing ECU to differentiation and binarizationprocessing to detect edges in the image and subjecting point sequencesof the detected edge (white line candidate point sequences) to Houghtransformation to extract approximated linear components.

Further, when, for example, a white line marked on the road surface isitself physically worn through or partially chipped away, or the whiteline is indistinct in the photographed image owing to low contrast suchas may occur during night driving, the white line cannot be accuratelyrecognized, and, as set out in Patent Reference 2, a technique has beenproposed for, in such a case, forming a window in the photographedimage, judging the degree of white line wear-off from the density of thewhite line candidate points in the window, and when wear-off is found,superimposing/compositing a white line candidate point sequencesextracted from a photographed image taken a given time earlier on/withthe white line candidate point sequences extracted from the currentwhite line image, determining a straight line approximating thecomposited white line candidate point sequences, and recognizing it asthe white line.

-   Patent Reference 1: Japanese Patent Publication No. Hei    6(1994)-42261-   Patent Reference 2: Japanese Laid-Open Patent Application No.    2001-236506

As mentioned above, in the technique set out in Patent Reference 2, achipped-away portion or an indistinct portion is covered or complementedby compositing (superimposing) the white line candidate point sequencesextracted from the current white line image with a white line candidatepoint sequences extracted from a photographed image taken a given timeearlier to elongate the white line candidate point sequences,determining a straight line approximating the elongated white linecandidate point sequences, and recognizing it as the white line.However, since the superimposing is conducted on the edge image afteredge detection, the accuracy of traffic lane marking line (white line)recognition is not necessarily satisfactory.

That is, since the size of a traffic lane marking line distant from thesubject vehicle in the photographed image is smaller than that of acloser traffic lane marking line, a case arises in which a distanttraffic lane marking line can not be recognized as edges. As a result,the recognition accuracy is not necessarily satisfactory, since pointsequences corresponding to a distant traffic lane marking line arechipped away and are not elongated in the composited image.

Therefore, an object of this invention is to overcome the aforesaiddrawbacks and provide a traffic lane marking line recognition system forvehicle configured such that point sequences corresponding to a distanttraffic lane marking line are prevented from being chipped away in acomposited image when compositing photographed images, thereby enablingaccurate and unerring recognition of the traffic lane marking line.

SUMMARY OF THE INVENTION

In order to achieve the object, according to a first aspect of theinvention there is provided a system for recognizing a traffic lanemarking line for a vehicle having image photographing means forphotographing a range including a road surface in a direction of travelof the vehicle, traffic lane marking line recognizing means configuredto be capable of recognizing at least a traffic lane marking line inbroken line on the road surface in a photographed image photographed bythe image photographing means, and image compositing means forcompositing a plurality of images photographed at different time pointsby the image photographing means to elongate the traffic lane markingline in the photographed images; wherein the image compositing meanscomposites the plurality of images in a processing stage in such amanner that no change will be imparted to at least a shape of thetraffic lane marking line in the photographed image in a traffic lanemarking line recognition processing of the traffic lane marking linerecognizing means.

Further, according to a second aspect of the invention there is provideda system for recognizing a traffic lane marking line for a vehiclehaving image photographing means for photographing a range including aroad surface in a direction of travel of the vehicle, traffic lanemarking line recognizing means configured to be capable of recognizingat least a traffic lane marking line in broken line on the road surfacein a photographed image photographed by the image photographing means bydetecting edges in the image and by conducting a Hough transformation onthe detected edges, and image compositing means for compositing aplurality of images photographed at different time points by the imagephotographing means to elongate the traffic lane marking line in thephotographed images; wherein the image compositing means composites theplurality of images in a processing stage prior to the edge detection ina traffic lane marking line recognition processing of the traffic lanemarking line recognizing means.

EFFECTS OF THE INVENTION

Since the traffic lane marking line recognition system for vehicleaccording to the first aspect is configured such that the plurality ofimages are composited in a processing stage in such a manner that nochange will be imparted to at least a shape of the traffic lane markingline in the photographed image in a traffic lane marking linerecognition processing, it becomes possible to detect a distant trafficlane marking line unerringly, and to elongate the traffic lane markingline in appearance, thereby improving recognition accuracy.

Further, segments of the traffic lane marking line are sometimes wornthrough or partially chipped away, and in some cases, segments aretemporality painted short in length as a stopgap measure duringconstruction work or for other such reason. In such cases, however, itbecomes possible to recognize the traffic lane marking line accurately.In addition, the direction of the traffic lane marking line can berecognized more accurately by the elongated traffic lane marking line,than by immediately recognizing broken lines (dotted lines) painted onroads, thereby enabling more accurate recognition of the traffic lanemarking line.

Since the traffic lane marking line recognition system for vehicleaccording to the second aspect is configured such that the plurality ofimages are composited in a processing stage prior to the edge detectionin a traffic lane marking line recognition processing, it also becomespossible to detect a distant traffic lane marking line unerringly, andto elongate the traffic lane marking line in appearance, therebyimproving recognition accuracy.

It should be noted that, the phrase “direction of travel of the vehicle”recited in claims is used to mean not only the forward direction of thevehicle when traveling forward, but also to recognize the road surfacebehind the vehicle photographed by an image photographing means forphotographing a range including the road surface behind the vehicle whenthe vehicle is traveling forward. Thus, the phrase “direction of travelof the vehicle” is used to mean the fore-aft direction of the vehicle.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic drawing showing the overall configuration of atraffic lane marking line recognition system for vehicle according to anembodiment of this invention;

FIG. 2 is a block diagram showing the operation of a control ECU shownin FIG. 1 in terms of inputs and outputs;

FIG. 3 is a flowchart showing the operation of the system shown in FIGS.1 and 2;

FIG. 4 is an explanatory view showing an edge image obtained by edgedetection processing shown in FIG. 3;

FIG. 5 is an explanatory view showing guidelines, i.e., linearcomponents corresponding to a white line or other such traffic lanemarking lines, which are obtained by Hough transformation processingshown in FIG. 3;

FIG. 6 is an explanatory view showing lane candidates (white linecandidate point sequences) determined by lane candidate detectionprocessing shown in FIG. 3;

FIG. 7 is an explanatory view showing an estimated location of a travellane of a subject vehicle determined by travel lane location estimationprocessing shown in FIG. 3;

FIG. 8 is a subroutine flowchart showing frame composition processingshown in FIG. 3;

FIG. 9 is an explanatory view showing a photographed image inputted atthe present time (current processing cycle) t that is to be a basis inpast frame selection processing shown in FIG. 8;

FIG. 10 is an explanatory view showing a photographed image inputted atthe time t-_(n) (earlier processing cycle) to be composited with thephotographed image inputted at the present time (current processingcycle) t in the past frame selection processing shown in FIG. 8;

FIG. 11 is an explanatory view showing a corrected image in vertical rowcorrection processing shown in FIG. 8;

FIG. 12 is an explanatory view showing a composited image obtained bycomposition processing shown in FIG. 8;

FIG. 13 is an explanatory view showing a predetermined region (roadsurface) of photographic range, which is to be detected by past framebrightness detection processing shown in FIG. 8;

FIG. 14 is an explanatory view showing the past frame brightnessdetection processing shown in FIG. 8;

FIG. 15 is a subroutine flowchart of brightness correction processingshown in FIG. 8;

FIG. 16 is an explanatory view showing characteristics of shutter speedsand irises with respect to the brightness of road surface, used in theprocessing of FIG. 15;

FIG. 17 is an explanatory view showing characteristics of amplifiergains with respect to the shutter speeds and irises, used in theprocessing of FIG. 15;

FIG. 18 is an explanatory view showing, as photographed by a camera,changes in the position and posture (angle) of the vehicle relative to atraffic lane marking line (lane), i.e., from exposure time t⁻² throughtime t⁻¹ up to time t₀ to explain horizontal pixel correction processingshown in FIG. 8;

FIG. 19 is a subroutine flowchart of the horizontal pixel correctionprocessing shown in FIG. 8;

FIG. 20 is a subroutine flowchart of the vertical row correctionprocessing shown in FIG. 8;

FIG. 21 is a subroutine flowchart of the processing of FIG. 19;

FIG. 22 is an explanatory view showing the processing of FIG. 21;

FIG. 23 is an explanatory view similarly showing the processing of FIG.21;

FIG. 24 is a subroutine flowchart of the processing of FIG. 19;

FIG. 25 is an explanatory view showing the processing of FIG. 24;

FIG. 26 is an explanatory view similarly showing the processing of FIG.24;

FIG. 27 is an explanatory view showing a composited image imparted withposition correction processing shown in FIG. 8;

FIG. 28 is an explanatory view showing a composited image in the case ofnot imparting the position correction processing shown in FIG. 8;

FIG. 29 is an explanatory view showing an image in the case where thecomposition is conducted prior to conducting edge detection processingin the composition processing of FIG. 8;

FIG. 30 is an explanatory view showing an image in the case where thecomposition is conducted after conducting edge detection processing inthe prior art;

FIG. 31 is an explanatory view showing a pattern matching method thatcan be used instead of the edge detection processing or Houghtransformation; and

FIG. 32 is an explanatory view showing an example of the patternmatching method shown in FIG. 31.

DETAILED DESCRIPTION INCLUDING BEST MODE OF CARRYING OUT THE INVENTION

The best mode for implementing the traffic lane marking line recognitionsystem for a vehicle according to this invention will be explained withreference to the attached drawings in the following.

First Embodiment

FIG. 1 is a schematic drawing showing the overall configuration of atraffic lane marking line recognition system for a vehicle according toan embodiment of this invention.

The symbol 10 in the drawing designates a camera equipped with an imagepickup device such as a CCD, C-MOS or the like and mounted inside apassenger compartment 12 a to face in the direction of travel of avehicle 12, which camera 10 photographs a region including the roadsurface in the direction of travel of the vehicle 12. An imageprocessing ECU (Electronic Control Unit) is accommodated in a case 10 aof the camera 10. The image processing ECU inputs images (pictures)outputted from the camera 10 representing information on the laneforward of the vehicle and is equipped with a hardware-implemented imageprocessing IC (Integrated Circuit) that performs image processingexplained later. The camera 10 and the image processing ECU correspondto the aforesaid image photographing means.

Note that in this embodiment “traffic lane marking line” means a roadmarking for separating vehicle passageways (lanes), such as a solid orbroken white or yellow line that is applied to the surface of the roadby painting, or cat's eyes or the like installed at intervals on theroad surface. A vehicle passageway partitioned by a traffic lane markingline or lines is called a “lane.”

A steering wheel 16 installed at a driver's seat in the passengercompartment 12 a of the vehicle 12 is connected to a rack shaft 20through a rack-and-pinion type steering gear and the rack shaft 20 isconnected to driven wheels 24 through tie rods 22. An electric powersteering (EPS) mechanism 30 including an electric motor 26 is disposedon the rack shaft 20 and the rack shaft 20 is reciprocated by rotationof the electric motor 26.

The driven wheels 24 and the free wheels (not shown) each has awheel-speed sensor (only two shown) 32 in the vicinity thereof thatproduces an output or signal once per predetermined angle of rotation,i.e., output indicative of the travel speed (vehicle speed) of thevehicle 12. A yaw rate sensor 34 is disposed at the middle of thevehicle 12 (near the rear axle) and produces an output or signalcorresponding to the yaw rate (angular velocity) about the vertical axis(gravity axis) at the center of gravity of the vehicle 12. Further, asteering angle sensor 36 is provided near the steering wheel 16 andproduces an output or signal corresponding to the amount of rotation ofthe steering wheel 16 manipulated by the driver, i.e., the steeringangle.

A control ECU (Electronic Control Unit) 40 similarly equipped with amicrocomputer is provided at a suitable location of the vehicle 12. Thecontrol ECU 40 inputs the outputs of the image processing ECU and theaforesaid sensors, calculates a steering force required for, inter alia,lane keep assist control for driving along the traffic lane or lanedeviation prevention control for preventing unintentional deviation fromthe traffic lane, and converts it into command values to output. Thecontrol ECU 40 is connected to an EPSECU 42 that controls the operationof the EPS 30. The EPSECU 42 is also equipped with a microcomputer,exchanges data with the control ECU 40, and operates the electric motor26 based on the command value outputted from the control ECU 40.

FIG. 2 is a block diagram showing the operation of the control ECU 40 interms of inputs and outputs.

As illustrated, the control ECU 40 inputs the output of the imagephotographing means composed of the camera 10 and the aforesaid imageprocessing ECU (designated by symbol 44) and the output of the yaw ratesensor 34 etc., and outputs lane keep assist control and other commandvalues to the EPSECU 42 by communication to operate the electric motor26. Further, although omitted in FIG. 1, meters and switches (SW) aredisposed near the driver's seat in the passenger compartment 12 a andthe particulars of the control performed by the control ECU 40 aredisplayed on the meters. Further, the driver's instructions to controlare inputted through the switches to be displayed on the meters and alsoto the control ECU 40.

FIG. 3 is a flowchart showing the operation of the system shown in FIGS.1 and 2. The program shown in the drawing is executed in the imageprocessing ECU 44 and control ECU 40 at predetermined time intervals of,say, 33 milliseconds.

This will be explained in the following: In step S10 (“step” is omittedhereinafter), the image pickup device of the camera 10 produces an imagesignal. The image pickup device of the camera 10 is provided with animaging region or range made up of n rows vertical×m columns horizontalpixels. On the factory production line, for example, the camera 10 hasits optical axis aligned in a predetermined direction including thetraffic lane ahead.

Next, in S12, frame composition processing is conducted, i.e., the imagesignal outputted from the image pickup device is inputted andcomposited. Specifically, the image signal inputted in the currentprocessing cycle and an image signal inputted and stored in the memoryof the image processing ECU 44 in an earlier processing cycle, in otherwords, the current image (frame) and a past image (frame) arecomposited. Note that since features that characterize this embodimentreside in the frame composition, this will be explained in detail below.Processing from S12 to S16 is performed by the hardware-implementedimage processing IC in the image processing ECU 44.

Next, in S14, edge detection processing including well-knowndifferentiation processing and ensuing binarization processing isconducted to produce from the inputted image an edge image like thatshown in FIG. 4, and next, in S16, linear components (shown by solidlines in FIG. 5) corresponding to white line or other such traffic lanemarking lines are discriminated by similarly well-known Houghtransformation.

Next, in S18, as shown in FIG. 6, a plurality of edge points (edge pointsequences; indicated as black bullets in the drawing) running along thelinear components are detected as lane candidates (white line candidatepoint sequences).

Next, in S20, when a plurality of candidates are present as lanecandidates (e.g., when, as shown in FIG. 6, three point sequences a, band c are present), which candidate among the multiple lane candidatesrepresents the travel lane of the subject vehicle (vehicle 12) isdiscriminated from the positional relationship between the subjectvehicle (vehicle 12) and the lane candidates, and the location of thetravel lane of the subject vehicle is estimated as shown in FIG. 7.

Next, in S22, vehicle control is implemented. That is, based on theestimated lane location, the command value for vehicle control, such asthe aforesaid lane keep assist control or lane deviation preventioncontrol, is outputted to the EPSECU 42 to control the operation of theelectric motor 26. Note that the processing of S12 to S16 is performedby the aforesaid hardware-implemented image processing IC in the imageprocessing ECU 44, the processing of S18 and S20 by the image processingECU 44, and the processing from S22 onward by the control ECU 40.

FIG. 8 is a subroutine flowchart showing the frame compositionprocessing of S12 in FIG. 3 (frame compositing of the currently acquiredimage (photographed image signal) and an image (photographed imagesignal) acquired and memorized at an earlier point in time).

The characterizing features of this embodiment reside in the variousframe composition shown in FIG. 8. Giving first a general explanationwith reference to this drawing, in S100 the earlier or past frame to becomposited is selected in accordance with the detected vehicle speed.

That is, the image to be composited with the photographed image inputtedat the present time (current processing cycle) t (shown in FIG. 9) isselected from among the multiple photographed images inputted in everyearlier processing cycle and stored in the memory of the imageprocessing ECU 44. Specifically, a photographed image, like that shownin FIG. 10, that was photographed and memorized at time t-_(n) isselected in accordance with the vehicle speed. Here, n means a discretesystem sampling time, and t_(−n) denotes an image photographed andinputted n cycles earlier, i.e., denotes an image photographed andstored at a different time point in the past.

In the ensuing S102, brightness correction of the selected earlier orpast frame is conducted. That is, the photographed image inputted at thepresent time and the image photographed at an earlier time point andselected in accordance with the vehicle speed in S100 were photographedat different time points, meaning that the brightness of thephotographed object (road surface) may differ owing to the effect ofshading and the like and that if composition should be conducted in astate where the brightness of the road surface portion of one image isbrighter than that of the white lines or other such traffic lane markinglines of the other image, there would be a risk of the white lines beingburied in the road surface in the composited image, and therefore, inorder to prevent this, processing is performed to match the brightnessof the image photographed at the earlier time point with the brightnessof the image photographed at the present time point such that thebrightness of the images at the present and earlier time points are madeequal. Note that the details thereof will be explained below.

In S104, the composition start point is determined. That is, it mayhappen that the posture of the vehicle 12 differ from the initial cameraparameter conditions (at the time of the optical axis alignment of thecamera 10) owing to vehicle loading or some other cause; in other words,a posture change may occur in the static pitching direction. As thisposture change may vertically shift the photographic range of the camera10, the composition start (reference) point is learned and corrected(determined) to take the amount of this shift into account at the timeof image composition. Since the region above the road surface isordinarily not needed for traffic lane marking line recognition, thisprocessing is performed by bringing the positions of the horizons at thepresent and earlier time points into registration.

Next, in S106, horizontal pixel correction is conducted. That is,depending on the image-photographing time points, the posture and angleof the vehicle 12 relative to a lane may differ between the present andearlier time points, so that cases may arise in which offset occurs bythe amount of such position and/or angle. In such a case, if the imagesare composited as they are, the horizontal position and angulardirection of the traffic lane marking line will of course also shift. Inorder to prevent this, the horizontal position and angular direction oftraffic lane marking line in the image photographed at the earlier timepoint is corrected by an amount proportional to the change in vehicleposture between the present and earlier time points. Note that thedetails thereof will be explained below.

Next, in S108, vertical row correction is conducted. That is, it isconceivable that the pitching condition of the vehicle 12 may differbetween the present and earlier time points, so the dynamic pitch anglevariation between the two time points is determined from the change inthe position of the horizon found from the images, and the vertical(up/down) direction in the image is corrected to obtain a correctedimage like that shown in FIG. 11. This will also be explained below.Note that, in contrast to S104, in which the static shift in the pitchis corrected, in S108 it is the dynamic shift thereof that is corrected.

Next, advancing to S110, the present time point's and corrected earliertime point's photographed images are composited to obtain an imagewherein, as shown in FIG. 12, the white lines and other such trafficlane marking lines elongated to a greater length, in appearance, than inthe images photographed at either of the two time points, as is evidentfrom a comparison with FIGS. 9 to 11. This completes the framecomposition processing shown in FIG. 8.

The features characterizing this embodiment will be successivelyexplained in the following; the first characterizing feature is, as hasbeen explained regarding S100 in FIG. 8, that the past frame to becomposited is selected in accordance with the vehicle speed (drivingspeed).

This will be explained in the following: In the prior art of PatentReference 2, the earlier image to be composited with the present imageis an image photographed a given time earlier. However, the traveldistance of the vehicle 12 differs with the vehicle speed (drivingspeed), which of course means that the photographic range of the camera(image pickup device) 10 similarly moves in the forward direction, sothat the positions of traffic lane marking lines in the image also movewith travel of the vehicle 12; in other words, the higher the vehiclespeed, the greater is the movement of the photographed traffic lanemarking line positions. Therefore, the amount and rate by which thetraffic lane marking lines are elongated in the composited image differsdepending on after how much of a time interval the earlier image isselected and composited with the present image.

Regarding traffic lane marking lines, there are known ones of brokenline configuration composed of periodically repeated white line segments(colored segments) and blank (uncolored segments of asphalt or thelike). From the aspect of durability, the white line segments of such atraffic lane marking line sometimes wear through or partially chip away,and in some cases, the white line segments are temporarily painted shortin length as a stopgap measure during construction work or for othersuch reason.

Further, while only natural, recognition of traffic lane marking linesby image recognition amounts to driving lane (subject vehicle lane)recognition, and since the direction of the traffic lane marking linecan be recognized more accurately in proportion as the length of thewhite line segments of the traffic lane marking lines is longer,composition is preferably done so as to elongate the white line segmentsin appearance, irrespective of presence/absence of the aforesaid whiteline segment wear and chipping. Taking this point into account, in thisembodiment the image concerned is selected from among images (pastframes) photographed earlier by a time interval determined in accordancewith the vehicle speed (driving speed). Note that this is not limited towhite lines but is also the same for yellow lines.

In view of what the inventors learned empirically, it is possible torecognize traffic lane marking lines with good accuracy and recognizethe subject vehicle's traffic lane unerringly provided that the lengthof (proportion accounted for by) the white line segments relative to thetotal length of the white line segments and blanks of the traffic lanemarking lines is equal to or greater than around 30%. The length of anelongated white line segment varies in accordance with the vehicle speedas mentioned earlier, and, therefore, the determination as to how manyframes earlier in the image-photographing cycle should be used is madein accordance with the controlled speed range of the vehicle controlsystems that utilize recognized traffic lane marking line data,including, inter alia, the lane keep assist control or lane deviationprevention control discussed at the beginning of the specification.

Specifically, the image-photographing cycle is 33.3 milliseconds, sothat if the vehicle speed (driving speed) is 60 km/h or faster, forexample, it suffices to use the image three frames earlier to secure therequired amount of white line segment elongation. Even at the image twoframes earlier, a vehicle speed of 90 km/h or faster suffices, althoughthe amount of white line segment elongation will be somewhatinsufficient, and even in the vehicle speed range of 60-90 km/h, theamount of white line segment elongation decreases but the lengthelongated is 1 meter or greater and the proportion of the total lengthaccounted for by the white line segments also increases to about 30%, sothat the traffic lane marking lines can be recognized with good accuracyto enable the traffic lane of the subject vehicle to be reliablyrecognized.

Therefore, at least one photographed image taken in the cycle matchingthe time interval determined in accordance with the detected vehiclespeed (driving speed) is selected as the photographed image to becomposited with a given (present) photographed image. Note that it isalso possible to switch the image used in the composition to two framesearlier or three frames earlier with the vehicle speed value impartedwith hysteresis at, for example, 105 km/h or 110 km/h, or, if thevehicle speed is 60 km/h or greater, to use only the image three framesearlier, because when using the image three frames earlier, the lengthof the white line segments relative to the total length of the whiteline segments and blanks of the traffic lane marking line becomes 30% orgreater.

Thus, in this embodiment the time of taking a given photographed imageamong the photographed images (specifically, the present image, stillmore specifically, the image photographed in the current traffic lanemarking line recognition processing cycle) is defined or determined asthe reference, and with respect to the given photographed image, thereis selected as the photographed image to be composited with thephotographed image at least one photographed image taken earlier by atime interval determined in accordance with the detected vehicle speed(driving speed) (more specifically, at least one photographed imagetaken in the cycle corresponding to the time interval determined inaccordance with the detected vehicle speed (driving speed)), so that,irrespective of how high or low the vehicle speed is, it becomespossible to optimally determine the amount of apparent elongation of thewhite line segments of the traffic lane marking lines or the elongationratio thereof and to enhance the recognition accuracy of the trafficlane marking lines to enable unerring recognition of the traffic lane ofthe subject vehicle.

Further, a configuration is adopted such that there is selected at leastone photographed image taken earlier by a time interval that is longerin proportion as the detected vehicle speed is lower. That is, since thetravel distance of the vehicle 12 becomes shorter with decreasingvehicle speed, an image photographed or inputted and memorized at a timepoint farther in the past is selected when the vehicle speed is low suchthat the amount of elongation or the elongation rate of the length ofthe traffic lane marking lines in the image after composition will notbe insufficient. On the other hand, since the travel distance of thevehicle 12 becomes longer with increasing vehicle speed, an imagephotographed or inputted and memorized at an earlier time point near thepresent time point is selected.

Further, there is selected at least one photographed image taken earlierby a time interval such that the length of the colored segments relativeto the total length of the white line segments (colored segments) andblanks (uncolored segments) of the traffic lane marking lines is equalto or greater than 30%, so that even when the traffic lane marking linesare worn through or chipped away, or short in length because ofconstruction work or the like, it becomes possible to optimallydetermine the amount of apparent elongation of the white line segmentsof the traffic lane marking lines or the elongation ratio thereof and toenhance the recognition accuracy of the traffic lane marking lines toenable unerring recognition of the traffic lane of the subject vehicle.

Further, the image on which the composition is based is made an imagephotographed in the current traffic lane marking line recognitionprocessing cycle, so that image composition can be performed based onup-to-date information. Further, there is selected at least onephotographed image taken in the cycle corresponding to the time intervaldetermined in accordance with the detected vehicle speed, so that imagecomposition can similarly be performed based on up-to-date information.

In this embodiment, the second characterizing feature is, as isexplained regarding S102 in FIG. 8, that brightness correction of theselected past frame is conducted.

This will be explained: In order to recognize the white line segments ofthe traffic lane marking lines with good accuracy, ordinarily, as shownin FIG. 13, the brightness of a predetermined region (road surface;hatched region in FIG. 13) of the photographic range of the camera 10 isdetected, the shutter speed and/or iris are adjusted based on thedetected brightness so as to enable clear detection of the white linesegments and, in addition, the photographed image signal outputamplifier gain and various photographic conditions are adjusted asnecessary.

Note that in this specification“brightness” is used in the meaning ofincluding all photographic conditions and the like, includingbrilliance, luminosity, density, and shutter speed, iris and imagesignal output amplifier gain and the like adjusted in accordance withthe detected image brightness.

In normal driving conditions, the brightness of the road surface changesfrom moment to moment because of the effect of various environmentalchanges, such as the shadows of buildings etc., wetting of the roadsurface, and strong sunshine. The aforesaid adjustment of photographicconditions is conducted for clearly detecting the white line segmentsdespite these brightness differences.

Consideration is now given to the compositing of the present and earlierimages in the case where the road surface brightness differs at theimage-photographing time points. Since the white line segments arebrighter than the remaining road surface (asphalt, concrete) in the sameimage, at the time of compositing, the brighter at the same point in theimage (same location) is assumed to be a white line segment andcomposition is conducted by selecting it.

As mentioned above, the shutter speed and the like are adjusted based onthe detected brightness to make the brightness between the photographedimages the same, but when variations of light and shade actually occuron the road surface, the shutter speed, iris and amplifier gain are notchanged in one stroke but, for preventing hunting and other reasons, aregradually changed, with the result that brightness does not become thesame between the photographed images but becomes somewhat different, sothat cases arise in which the white line segments in one photographedimage become darker than the road surface (asphalt portions) in theother photographed image.

FIG. 14 is an explanatory view showing image brightness correctionprocessing in this embodiment. Note that the three graphs marked a, b, cshown on the right side of the drawing are diagrams schematicallyindicating the brightnesses of the locations of the white line segmentsand road surface (asphalt) designated by broken lines in the images onthe left side of the drawing. The annotation above graph a in thedrawing indicates that the projecting part of the pulse-like waveform isthe white line segment and the parts on opposite sides thereof the roadsurface. Therefore, graph a in the drawing shows that in the images A, Bon the left side, the road surface of B is brighter than the white linesegment of A.

A in FIG. 14 is an image photographed and memorized at an earlier timepoint, and B is the image photographed at the present time point. Asshown by graph a on the right, the brightnesses of the images are suchthat the road surface of image B is brighter than the white line segmentof image A (broken-line portion in the figure). As explained below, inthis embodiment, as shown at A′ of FIG. 14, position correction of theimages is conducted in accordance with the change in the vehicleposition between the present time point and the earlier time point.

At this time point, there is no change between A and A′, and if at thisstage, as indicated by the broken line with accompanying remark “W/OCORRECTION” in FIG. 14, the two images are composited, then, asindicated by the graph a on the right, the road surface portion of imageB comes to be selected because the road surface (asphalt) of image B isbrighter than the white line segment of image A, which gives rise to thedisadvantage that the image becomes like the image in C′ of FIG. 14 inwhich the white line segment of image A (A′) has disappeared.

In view of this point, in this embodiment the brightness of thephotographed image is detected from the brightness or the like of apredetermined road surface region in the photographic range (hatchedregion in FIG. 13) and, taking one image among a plurality ofphotographed images as the reference, correction is carried out so as tomake the brightness the same in the plurality of photographed images. Inthis case, the brightness can actually be detected or be detectedexpediently. Specifically, the brightness ratio of the two images isdetermined from the photographic conditions when the images were takenat the earlier time point and the present time point and using theresult to make the brightness of the two images the same, therebyeliminating the aforesaid disadvantage. Note that the brightness of thecomposited images can also be made equal by using the brightnessdifference rather than the ratio.

FIG. 15 is a subroutine flowchart of the brightness correctionprocessing of S102 of FIG. 8.

This will be explained in the following: First, in S200, S202, thephotographic conditions at the time of image-photographing, i.e., thephotographic conditions comprising the shutter speeds S0, S1, irises I0,I1 and amplifier gains G0, G1 indicative of the image signalamplification factors are memorized. The detected photographed imagebrightness, specifically the brightness of a predetermined region of theroad surface (hatched region in FIG. 13) in the photographic range ofthe image photographing means, is detected from the pixel density or thelike of the region, and the shutter speed is determined (defined) basedon the detected road surface brightness in accordance with thecharacteristic shown in FIG. 16, such that the sensitivity of the imagepickup device (pixels) of the camera 10 is adjusted.

Note that, as is well known, image brightness can be adjusted also bythe iris adjustment because it varies the amount of light taken in, asshown in FIG. 16. Further, when the ambience is dark and the imagedensity adjusted by one or both of the shutter speed and iris is stillinsufficient, the brightness can be adjusted by the amplification factorn of the amplifier gain. It is with this in mind that in S200, S202 theiris and amplifier gain are also stored in memory in addition to theshutter speed.

Next, in S204, the brightness ratio between the present time point andearlier time point images is calculated based on the memorizedphotographic condition using the equation shown there, and in S206, thebrightness of the earlier time point image is multiplied by thecalculated brightness ratio to carry out correction for making thebrightness of the earlier time point image the same as the brightness ofthe present time point image.

More specifically, the brightness ratios of a plurality of photographedimages are detected based on at least one of the shutter speed and irisand the brightness is corrected based on at least one of the ratio anddifference of one or both of the shutter speed and iris at theimage-photographing time point of each of the photographed images, morespecifically, based on both ratios thereof. Further, the brightnessratios of the multiple photographed images are also detected based onthe amplification factors.

Regarding the brightness, note that the brightness of the predeterminedregion of the road surface indicated by the hatched region in FIG. 13 isdetected from the pixel density of the region, and the detectedbrightness is corrected based on at least the respective ratios of theshutter speeds and irises at the image-photographing time points of thedetermined multiple photographed images.

As a result, in the processing of S110 of FIG. 8, an earlier time pointimage corrected in brightness and the present time point image arecomposited. Note that in the processing of S110, because the white lineor other such traffic lane marking line is ordinarily brighter than thesurrounding asphalt or other such road surface, at the time ofcomposition there is selected from among the multiple images to becomposited one whose pixels constituting the same area (given area)among multiple areas are brighter in brightness, i.e., that is brighterat the individual pixel level.

When the brightness is corrected by this processing, then, as shown bygraph b on the right side of FIG. 14, the brightnesses of the roadsurface in the earlier time point image (broken line portion of image A″of FIG. 14) and in the present time point image (broken line portion ofimage B of FIG. 14) become the same. Therefore, as shown in C of FIG.14, when the two images are composited, the white line segment is notburied in the road surface and there can be obtained an image whereinthe white line segment is suitably elongated.

As set out in the foregoing, this embodiment is configured to detect thebrightnesses of the photographed images photographed by the imagephotographing means, to perform correction to make the brightness inmultiple photographed images the same, taking one image among themultiple photographed images as the reference, and, after the brightnesshas been corrected, to composite the multiple images, whereby, as aresult of the brightness of the two images being the same, the whiteline segment in one of the images at the earlier time point and presenttime point does not become buried in the road surface of the other imageand the white line segment can be favorably elongated. Therefore, thetraffic lane marking line recognition accuracy can be enhanced and thetraffic lane of the subject vehicle can be reliably recognized.

Further, since the image constituting the reference for composition isthe image of the current traffic lane marking line recognitionprocessing cycle, the images can be composited while making brightnessequal based on up-to-date information to achieve improved traffic lanemarking line recognition accuracy, thereby enabling reliable recognitionof the subject vehicle's traffic lane. Further, by detecting thebrightness of a predetermined region of the road surface, the brightnessdetection can be performed using a common yardstick to improve thetraffic lane marking line recognition accuracy.

Further, brightness detection and correction are simple because thebrightness detection and correction are performed based on at least oneof the ratio and difference of one or both of the shutter speed and irisat the image-photographing time point of each of the multiplephotographed images and on the amplifier gain (amplification factor).Since the correction is made so as to make the brightnesses equal at thesame location in multiple images, the brightness can be corrected withstill better accuracy.

Further, a configuration is adopted wherein multiple photographed imagesare composited by selecting from among the pixels constituting the samearea of the multiple images that is brighter in brightness, andtherefore, the traffic lane marking line can be still more accuratelyrecognized from the composited image, whereby the traffic lane of thesubject vehicle can be recognized still more reliably.

In this embodiment, the third characterizing feature is, as is explainedregarding S106 and S108 in FIG. 8, that horizontal pixel correction andvertical row correction are conducted.

This will be explained: The posture (angle) of the vehicle 12 relativeto the traffic lane marking line and the distance to the traffic lanemarking line in the horizontal direction sometime differ between thepresent time point and earlier time point, so that the angular directionof the vehicle 12 with respect to the horizontal direction and trafficlane marking line is shifted between the images photographed at the twotime points, which of course means that if the images are composited asthey are, an offset condition will occur in the horizontal position andangle of the traffic lane marking line.

In order to prevent this, it is necessary to correct the horizontalposition and angular direction of the traffic lane marking line in theimage at the earlier time point by an amount corresponding to the changein the vehicle posture between the present and earlier time points, andfurther, since it is conceivable that the pitching condition of thevehicle 12 may differ between the present time point and earlier timepoint, it is necessary to determine the pitch angle change between thetwo time points from the change in the horizon position and the likeobtained from the images and correct vertical direction position in theimages.

On this point, the prior art taught by Patent reference 2 merely detectsthe amount of vehicle yaw rate change, i.e., presence/absence of vehiclerotation (turning) movement, and imparts correction to the image incorrespondence to the amount of rotation movement, if any, so that nocorrection of the horizontal position is carried out and, therefore,there is a disadvantage in that the traffic lane marking line is notsuitably elongated in the composited image. Further, the technology ofPatent reference 2 is disadvantageous in that it does not take pitchchange into consideration.

FIG. 18 is an explanatory diagram showing, as taken by the camera 10,changes in the position and posture (angle) of the vehicle 12 relativeto a traffic lane marking line (lane), i.e., from exposure time t⁻²through time t⁻¹ up to time t₀. Here, exposure time t₀ is the presenttime point, time t⁻¹ is the earlier time point, and time t⁻² is theimage-photographing time point of the image used for compositing theimages taken at time t₀ and t⁻¹ and is farther in the past than the timeof the earlier time point t⁻¹.

In this embodiment, the traffic lane marking lines on the left and rightof the vehicle 12 in FIG. 18 (the lane) are recognized, and taking thecenter thereof as the reference point, the point sequences that arecontinuums of these points and the line segments connecting them (shortbroken lines in FIG. 18) are detected and used as lines along which thevehicle is to travel in the case of, for example, lane keep assistcontrol. Note that since the traffic lane marking lines in FIG. 18 areones obtained by image photographing at the earlier time point (timet⁻²), the point sequences are also the point sequences at time t⁻². Thedistances between point sequences and vehicle positions (distancesbetween the traffic lane marking lines and the vehicle 12) in thevehicle positions at exposure times t₀, t⁻¹ are designated L₀ and L₁ andthe angle between the line segments connecting the point sequences anddirection of the vehicle 12 (the fore-aft direction angle of the vehicle12 along the traffic lane marking lines) as θ₀ and θ₁.

In order to composite the image photographed and memorized at exposuretime t⁻¹, which is the earlier time point, with the image photographedat the present time point, i.e., the exposure time t₀, taking thetraffic lane marking lines (lane), point sequences and line segmentsobtained at exposure time t⁻² as the references, the distances L₀, L₁and angles θ₀, θ₁ are determined, the deviations or changes between L₀and L₁ and between θ₀ and θ₁ are determined, and once the changes in thephotographed images caused by the changes in the relative position andangle of the vehicle 12 relative to the traffic lane marking linesbetween times t₀ and t⁻¹ have been corrected, the images at the two timepoints are composited.

FIG. 19 is a subroutine flowchart of the horizontal pixel correctionprocessing of S106 of FIG. 8.

This figure shows the processing for determining the lateral positiondeviation between exposure times t₀ and t⁻¹ (amount of past (time t⁻¹)lateral vehicle movement relative to present time t₀) ΔL and the angledeviation relative to the traffic lane marking lines or the referenceline therebetween (past (time t⁻¹) angle deviation relative to presenttime t₀) Δθ, and then determining the amount of horizontal (lateral)pixel position correction in the images from the determined deviations.

Although the details will be explained below, in S300, the deviationsΔL, Δθ are determined, and in S302 the pixel position correction amounton the image plane at compositing is determined from the deviations ΔL,Δθ, and correlation of the camera coordinate system (U, V) and theactual plane coordinate system (X, Y) is performed as pre-compositinghardware-based processing. This processing of S302 will be explained indetail below.

FIG. 20 is a subroutine flowchart of the vertical row correctionprocessing of S108 of FIG. 8.

First, in S400, the pitch direction change Δθpit between exposure timest₀ and t⁻¹ is determined, and in S402 the amount of vertical rowposition correction in the image plane at the time of composition isdetermined based on the change thereof, and correlation of the cameracoordinate system (U, V) and the actual plane coordinate system (X, Y)is also performed as pre-compositing hardware-based processing.

FIG. 21 is a subroutine flowchart of the processing of S300 in FIG. 19,and FIGS. 22 and 23 are explanatory diagrams thereof.

Explanation will be made with reference to FIGS. 21 to 23. In FIG. 22,the coordinate axes for trajectory estimation are coordinate axes whoseX axis is set to the fore-aft-direction center axis of the vehicle(vehicle center axis) at a given time in the past. In order to determinethe lateral or horizontal direction change in the position of thevehicle at exposure time t⁻¹ in this coordinate system, taking thevehicle position at exposure time t⁻² as the reference, consideration isgiven to a detection result coordinate system X, Y whose origin (x,y)=(0, 0) is set at the vehicle (center) position at exposure time t⁻²and whose X axis is set to the vehicle center axis direction (fore-aftdirection).

The position and direction of the vehicle 12 at exposure time t⁻¹ areconverted into values on the time t⁻²-based X, Y coordinate system usingthe estimated trajectory (locus) of the vehicle between exposure timest₀ and t⁻¹ determined from the outputs of the wheel-speed sensor 32 andthe yaw rate sensor 34. At this time, the coordinates of the position ofthe vehicle 12 on the detection result coordinate system X, Y atexposure time t⁻¹ are (x, y). In (1) of S500 in FIG. 21, the coordinates(x, y) of the position of the vehicle at exposure time t⁻¹ aredetermined.

In (2) of S500, as shown in FIG. 22, the distance L⁻¹ between point (x,y_(p)) directly sideways from the coordinates (x, y), i.e., the samevalue in the X axis and directly sideways from the coordinates (x, y) onthe straight line connecting the point sequences P_(k), P_(k+1) (atwhich point the vehicle is positioned at exposure time t⁻¹) among thepoint sequences 1, P_(k), P_(k+1) that were determined at exposure timet⁻², and the aforesaid coordinates (x, y) is determined by subtractionas L⁻¹=y_(p)−y.

Similarly, in (3) of S500, the distance L₀ between the vehicle positionat exposure time t₀ and the straight line connecting the point sequencesis determined taking exposure time t⁻² as the reference. Note that inthe interest of simplicity, the vehicle position at exposure time t₀,point sequence P_(k+2), and so on are omitted in FIG. 22.

Then, in (4) of S500, the lateral direction position deviation ΔLbetween exposure times t₀ and t⁻¹ is determined by the subtraction asL₀−L⁻¹. The sign of ΔL of course becomes positive or negative dependingon the vehicle position at the exposure time points.

Next, the angle deviation Δθ of the vehicle relative to the referenceline between exposure times t₀ and t⁻¹ i.e., the point sequences and thestraight line connecting them, is determined.

First, in (1) of S502 in FIG. 21, as shown in FIG. 23, the angledeviation in the trajectory estimation coordinate system shown in FIG.22 can be determined from difference in the estimated trajectory valuesθ at exposure time t⁻¹ and exposure time t₀. That is, the angulardeviation is determined as θ₀−θ⁻¹, when defining the estimatedtrajectory value θ in the trajectory estimation coordinate system atexposure time t₀ as θ₀ and the estimated trajectory value at exposuretime t⁻¹ as θ⁻¹.

Further, when the straight lines, determined at exposure time t⁻²,connecting the point sequences P_(i), P_(i+1), P_(i+2) where the vehicleis positioned at two times, i.e., exposure times t₀ and t⁻¹, differ sothat the straight lines are not a single straight line, it is necessaryto determine the angle deviation θ_(R) between the straight lines, asshown in FIG. 23. Of course if the straight lines are a single straightline or the vehicle position at the two time points is within the rangeof the same straight line, the angle deviation θ_(R)=0.

In (2) of S502 in FIG. 21, the point sequence photographed and detectedat exposure time t⁻² is used to determine the angle deviation θ_(R) ofthe straight line where the vehicle is positioned at exposure times t⁻¹,t₀. Then in (3) of S502, the angle deviation Δθ is calculated asΔθ=θ₀−θ⁻¹−θ_(R).

FIG. 24 is a subroutine flowchart showing the processing of S302 in FIG.19 and FIGS. 25 and 26 are explanatory diagrams thereof.

FIG. 25 is what is obtained by simplifying and depicting therelationship of times t⁻¹ and t₀ with the vehicle on the basis of theposition deviation ΔL and angle deviation Δθ relative to the referenceline determined earlier in FIGS. 21 and 22. The relationship between thedistance in the vehicle fore-aft direction in the actual plane'scoordinate system, i.e. the lateral distance Y at X and a given X, is,from Δθ and ΔL,y=x·tan Δθ+ΔL.

In S600 of FIG. 24, the corrective distances Y_(5 m) and Y_(30 m)required during horizontal direction compositing in the actual plane atdistances X=5 meters and X=30 meters are determined from theabove-mentioned equation. Here, 5 meters and 30 meters are examples andcan be appropriately defined from the breadth of the photographic rangeof the camera, and the like.

Next, in S602, the line positions V_(5 m) and V_(30 m) in the imageplane corresponding to the distances X=5 m and X=30 m in the actualplane are determined. The image plane coordinate system is shown in FIG.26. The V axis corresponds to the distance direction X axis in theactual plane, the top in the vertical direction is made the origin, andthe U axis corresponds to the lateral or horizontal Y axis. These linepositions V_(5 m) and V_(30 m) are determined with consideration to thecorrected value obtained by learning the pitch direction shift explainedin S104 of FIG. 8.

Next, in S604, the lateral pixel positions U_(5 m) and U_(30 m) in theimage plane that correspond to the lateral corrective distances Y_(5 m)and Y_(30 m) obtained in S600 are determined.

Next, in S606, the straight line passing through the two points of theimage plane coordinates (U5 m, V5 m) and (U30 m, V30 m) obtained in S606is determined. In this embodiment, the composition positions at the timeof compositing images photographed at different time points aredetermined such that six points x0, x1, . . . , x6 on the V axis of theimage plane fall on this straight line. That is, in order to correct theposition and angle change relative to the reference line in the actualplane at time points t⁻¹, t₀, the straight line indicating the positionand angle of the vehicle at time t⁻¹ in the actual plane (coincidentwith the X axis as the reference) is made equal to the straight lineindicating the position and angle at time t₀ shown in FIG. 25. Note thatthis determination of the compositing position can be rapidly processedbecause it is processed in the hardware-implemented image processing IC.

FIGS. 27 and 28 are diagrams showing image-composited images impartedwith the position correction processing of this embodiment andimage-composited images in the case of not imparting the positioncorrection.

When vehicle lateral direction position and angle correction relative tothe traffic lane marking lines is imparted, as in FIG. 27, the trafficlane marking lines assume an elongated form as shown in image C of thesame figure, while when correction is not imparted, it can be seen that,as in FIG. 28, the traffic lane marking lines photographed at theearlier and present time points are composited in an offset form and thetraffic lane marking lines are not elongated.

In this embodiment, the fore-aft direction angles θ of the vehicle 12relative to the traffic lane marking lines and the distances L betweenthe traffic lane marking lines and the vehicle 12 in multiplephotographed images are determined, the angle and position deviations(Δθ, ΔL) between the respective photographed images are determined, andthe multiple images are composited after having been corrected such thatthe angle and distance of the vehicle relative to the traffic lanemarking line are equal to each other in the images based on thedetermined angle and position deviations, whereby the traffic lanemarking lines can be reliably elongated in appearance in the compositedimage.

Further, the pitch angles of the vehicle 12 at image-photographing timepoints t⁻¹ and t₀ of multiple photographed images are detected todetermine the pitch angle change (Δθpit) between the photographedimages, and the multiple images are composited after being correctedsuch that the pitch angles are equal to each other in the images basedon the determined pitch angle change, whereby the traffic lane markinglines can be still more reliably elongated in appearance.

Note that, instead of the foregoing, it is acceptable to composite byadjusting the compositing positions between the multiple images based onthe ascertained angle and position deviations to the condition of beingphotographed at the same vehicle angle and position relative to thetraffic lane marking lines. That is, rather than move the imagesthemselves as set out in the foregoing, it is acceptable to adjust thecompositing positions (superimposition positions).

To explain with reference to FIG. 27, if one or the other of the earlieracquired image A and the presently acquired image B is itself shifted bythe amount of the aforesaid deviations in lateral direction position andangle between the two time points and superimposed on and compositedwith the other image, the same result as that of the aforesaidcorrection is obtained. That is, considering the correction of thelateral direction position, it suffices to make point q of image B inFIG. 27, which is laterally offset from point p of image A by the amountof the lateral direction position correction, fall on point p. Althoughthere is almost no change in angle between the two time points, whenthere is an angle change, it suffices to composite after the image isrotated in accordance with the amount of angle change. Further, also ina case where there is a pitch angle change, it suffices to composite byshifting one of the images in the vertical direction of FIG. 27 by anamount corresponding to the change. By doing this, the same effect asthat of the aforesaid image correction can be obtained.

In this embodiment, the fourth characterizing feature is that in thecomposition processing of S110 in FIG. 8, the present time point andearlier time point images are composited at the original image stagebefore edge detection processing.

This will be explained: FIG. 29 shows images prior to conducting edgedetection processing, in other words in the case where the present timepoint and earlier time point images are composited at the original imagestage, and FIG. 30 shows images after conducting edge detectionprocessing, in other words in the case where composition of the presenttime point and earlier time point is conducted in the so-callededge-image condition as in prior art of Patent Reference 2.

In the prior art processing of FIG. 30, as in images A, B thereof, thelength of the traffic lane marking lines far distant from the subjectvehicle is short in the images, i.e., when the present time point andearlier time point images are edge- detection-processed individually,they are not recognized as edges because the size in the images issmall, and as a result, as in images a, b of FIG. 30, a condition ofthere being nothing at all in the far distance of the edge image arisesas shown enclosed by a broken line.

Next position correction processing is performed based on the vehicleposture change to obtain the position-corrected image shown in image a′in FIG. 30, and even if the present time point and earlier time pointedge images are then composited, the result has no edges correspondingto far distant traffic lane marking lines visible in the originalimages, as in the region enclosed by a broken line in image D of FIG.30.

In contrast to this, as shown in FIG. 29, in the case where the earliertime point image (image A in FIG. 29) is position-corrected based on thevehicle posture change (image A′ in FIG. 29) and image brightnesscorrection is performed (image A″ in FIG. 29), and the present timepoint and earlier time point images are composited at the original imagestage prior to performing edge detection processing, the short trafficlane marking lines in the far distance in the respective original imagesare elongated and become long, as in image C in FIG. 29, and even ifedge detection processing is performed thereafter, can be detected asedges, as in the region enclosed by a solid line in image D of FIG. 29.

Thus, in this embodiment, the configuration is adopted such that thephotographed images are composited in a processing stage in such amanner that no change will be imparted to at least the shape of thetraffic lane marking lines in the photographed image in the traffic lanemarking line recognition processing, more specifically, before edgedetection, i.e., such that the present time point and earlier time pointimages are composited at the original image stage prior to edgedetection processing, so that distant traffic lane marking lines can bedetected, thereby making it possible to extend the traffic lane markinglines in appearance and improve recognition accuracy.

Note that although it has been defined as being before edge detectionprocessing, it suffices for the shape and profile of the traffic lanemarking lines in the photographed original images to be in a retainedcondition and since, as has been mentioned above, edge detectionprocessing is ordinarily constituted of differentiation processing,binarization processing and edge detection, the effect of the presentinvention can be obtained if compositing is done before binarizationprocessing even if not before edge detection processing, i.e., beforedifferentiation processing.

Note that although this invention has been explained taking traffic lanemarking line recognition by edge detection processing and Houghtransformation as an example, this invention is not limited thereto andby elongating the traffic lane marking line segments with respect toanother method, e.g., a method using pattern matching such as shown inFIG. 31 or set out Japanese Laid-Open Patent Application No. Hei8(1996)-297728, more accurate pattern matching is possible, which iseffective. Such pattern matching is shown in FIG. 32.

That is, a method is known wherein the traffic lane marking lines areestablished beforehand by defining shape patterns in an imagephotographed by an on-board camera as multiple templates such as shownin FIG. 32, sequentially applying these to the photographed image asshown in the middle diagram of FIG. 31, determining the degree ofcoincidence of the image and templates, and, as in the bottom diagram ofFIG. 31, and, from the coincidence degree distribution, defining thelines formed by the points with the highest degree of coincidence as thetraffic lane marking lines; and also in such pattern matching, thepresent invention can elongate the traffic lane marking lines to improvethe recognition accuracy of the traffic lane marking lines.

Further, in this embodiment, although the number of images to becomposited in the processing of S100 is two, it can as necessary be alarger plural number. In addition, in this embodiment, selection is madein accordance with vehicle speed from among images periodicallyphotographed, but it also acceptable, for example, to provide two ormore image photographing means and select in accordance with the vehiclespeed from among images not photographed in the same common cycle.

Further, in the correction of S108, it is acceptable to make thecorrection by, instead of using the change in the horizon obtained fromthe images, installing a pitch angle sensor at an appropriate locationof the vehicle 12 and determining dynamic offset from the outputthereof.

Further, although a configuration is adopted whereby the camera 10photographs a range including the road surface ahead of the vehicle, itis of course acceptable to adopt a configuration provided with an imagephotographing means (camera) for photographing a range including theroad surface behind the vehicle when the vehicle 12 is travelingforward, wherein the traffic lane marking lines behind the vehicle arerecognized from the photographed images taken by the image photographingmeans. Therefore, in this specification “direction of travel of thevehicle” is used to mean the fore-aft direction of the vehicle.

As stated above, the embodiment is thus configured to provide a systemfor recognizing a traffic lane marking line for a vehicle 12 includingimage photographing means (camera 10, image processing ECU 44) forphotographing a range including a road surface in a direction of travelof the vehicle, traffic lane marking line recognizing means (imageprocessing ECU 44, S12 to S16) configured to be capable of recognizingat least a traffic lane marking line in broken line on the road surfacein a photographed image photographed by the image photographing means,and image compositing means (image processing ECU 44, S12) forcompositing a plurality of images photographed at different time pointsby the image photographing means to elongate the traffic lane markingline in the photographed images; wherein the image compositing meanscomposites the plurality of images in a processing stage in such amanner that no change will be imparted to at least a shape of thetraffic lane marking line in the photographed image in a traffic lanemarking line recognition processing of the traffic lane marking linerecognizing means (S12).

Further, the embodiment is configured to provide a system forrecognizing a traffic lane marking line for a vehicle 12 including imagephotographing means (camera 10, image processing ECU 44) forphotographing a range including a road surface in a direction of travelof the vehicle, traffic lane marking line recognizing means (imageprocessing ECU 44, S12 to S16) configured to be capable of recognizingat least a traffic lane marking line in broken line on the road surfacein a photographed image photographed by the image photographing means bydetecting edges in the image and by conducting a Hough transformation onthe detected edges, and image compositing means (image processing ECU44, S12) for compositing a plurality of images photographed at differenttime points by the image photographing means to elongate the trafficlane marking line in the photographed images; wherein the imagecompositing means composites the plurality of images in a processingstage prior to the edge detection (S14) in a traffic lane marking linerecognition processing of the traffic lane marking line recognizingmeans.

INDUSTRIAL APPLICABILITY

According to this invention, since it is configured such that theplurality of images are composited in a processing stage in such amanner that no change will be imparted to at least a shape of thetraffic lane marking line in the photographed image in a traffic lanemarking line recognition processing, it becomes possible to detect adistant traffic lane marking line unerringly, and to elongate thetraffic lane marking line in appearance, thereby improving recognitionaccuracy. Further, segments of the traffic lane marking line aresometimes worn through or partially chipped away, and in some cases,segments are temporality painted short in length as a stopgap measureduring construction work or for other such reason. In such cases,however, it becomes possible to recognize the traffic lane marking lineaccurately. In addition, the direction of the traffic lane marking linecan be recognized more accurately by the elongated traffic lane markingline, than by immediately recognizing broken lines (dotted lines)painted on roads, thereby enabling to provide a traffic lane markingline recognition system for vehicle that can recognize the traffic lanemarking line more accurately.

Although there have been described what are the present exemplaryembodiments of the invention, it will be understood that variations andmodifications may be made thereto within the spirit and scope of theappended claims.

1. A system for recognizing a traffic lane marking line for a vehiclecomprising: image photographing means for photographing a rangeincluding a road surface in a direction of travel of the vehicle;traffic lane marking line recognizing means configured to be capable ofrecognizing at least a traffic lane marking line in broken line on theroad surface in a photographed image photographed by the imagephotographing means; and image compositing means for compositing aplurality of images photographed at different time points by the imagephotographing means to elongate the traffic lane marking line in thephotographed images; wherein the image compositing means selects atleast one of the plurality of images and composites the plurality ofimages in a processing stage in such a manner that no change will beimparted to at least a shape of the traffic lane marking line in thephotographed image in a traffic lane marking line recognition processingof the traffic lane marking line recognizing means.
 2. A system forrecognizing a traffic lane marking line for a vehicle comprising imagephotographing means for photographing a range including a road surfacein a direction of travel of the vehicle; traffic lane marking linerecognizing means configured to be capable of recognizing at least atraffic lane marking line in broken line on the road surface in aphotographed image photographed by the image photographing means bydetecting edges in the image and by conducting a Hough transformation onthe detected edges; and image compositing means for compositing aplurality of images photographed at different time points by the imagephotographing means to elongate the traffic lane marking line in thephotographed images; wherein the image compositing means selects atleast one of the plurality of images and composites the plurality ofimages in a processing stage prior to the edge detection in a trafficlane marking line recognition processing of the traffic lane markingline recognizing means.
 3. A vehicle comprising: image photographingmeans for photographing a range including a road surface in a directionof travel of the vehicle; traffic lane marking line recognizing meansconfigured to be capable of recognizing at least a traffic lane markingline in broken line on the road surface in a photographed imagephotographed by the image photographing means; and image compositingmeans for compositing a plurality of images photographed at differenttime points by the image photographing means to elongate the trafficlane marking line in the photographed images; wherein the imagecompositing means selects at least one of the plurality of images andcomposites the plurality of images in a processing stage in such amanner that no change will be imparted to at least a shape of thetraffic lane marking line in the photographed image in a traffic lanemarking line recognition processing of the traffic lane marking linerecognizing means.
 4. A vehicle comprising: image photographing meansfor photographing a range including a road surface in a direction oftravel of the vehicle; traffic lane marking line recognizing meansconfigured to be capable of recognizing at least a traffic lane markingline in broken line on the surface of road in a photographed imagephotographed by the image photographing means by detecting edges in theimage and by conducting a Hough transformation on the detected edges;and image compositing means for compositing a plurality of imagesphotographed at different time points by the image photographing meansto elongate the traffic lane marking line in the photographed images;wherein the image compositing means selects at least one of theplurality of images and composites the plurality of images in aprocessing stage prior to the edge detection in a traffic lane markingline recognition processing of the traffic lane marking line recognizingmeans.